Many types of architectures for computer-based editing are found in the industry to manipulate or to edit sequences of video, audio or the like.
These conventional architectures may be divided into three broad categories: timeline, process nodes and animation curves architectures.
Timeline Architectures
Timeline editing involves the ordering of video and/or audio segments, each representing a single scene or a logical unit of video or audio data, into a multimedia sequence.
FIG. 1 is a schematic view illustrating a display screen of a timeline editor. Timeline editors typically include a stack of horizontal bars 10-18 sub-divided into segments 20 which run along an x-axis 22 which is calibrated into time units.
Horizontal bar 10 is associated with a video channel while each horizontal bar 12 to 18 is associated with an audio channel. The video and audio channels are mapped onto particular playback devices. The order of the segments 20 on each horizontal bar 10 to 18 regulates the order in which these segments will be played on the playback device associated with each particular horizontal bar.
One of the advantages of timeline editing is that both the temporal arrangement of all the elements in a multimedia sequence as well as their relationship to the playback device is obvious to the user. Segments can be intuitively placed and reordered in time sequence as well as moved between playback channels.
One drawback of timeline editing is that it is impossible to modify the segments other than to modify their duration.
Process Node Architectures
Process node editing typically involves an arc and node graph, hereinafter referred to as a process node graph, in which each node can be either a source (input), a transformation (filter), or a sink (output). Each arc of a process node graph shows the passage of video or audio data from one node to another.
The purpose of the process node graph is to represent the construction history of a multimedia sequence. By following the arcs from the source(s) to the sink, the user may retrace the evolution of the segments being transformed or otherwise modified by each node.
FIG. 2 of the appended drawings shows a simple process node graph illustrating the transformation of video segments. Two sources 24 and 26, and three transformations 28, 30 and 32 are used to produce a desired sink 34.
The segment of source 24 is supplied to an input 36 of the first transformation 28, which is a color corrector, through an arc 38. The result of the first transformation 28 is illustrated in the color corrected segments 40 and 42 respectively connected to outputs 44 and 46 is the first transformation 28 through arcs 48 and 50.
The color corrected segment 40 is supplied to an input 52 of the second transformation 30, which is a horizontal flip, through an arc 54. The result of the second transformation 30 is illustrated in the flipped color corrected segment 56 connected to an output 58 of the second transformation 30 via an arc 60. The flipped color corrected segment 56 is supplied to a first input 62 of the third transformation 32 via an arc 64.
The color corrected segment 42 is supplied to a second input 66 of the third transformation 32 via an arc 68.
The segment of the second source 26 is supplied to a third input 70 of the third transformation 32 via an arc 72.
The function of the third transformation 32, called a keyer, is to superpose a portion of the segment supplied to its first input 62 to the segment supplied to its third input 70. The portion superposed is defined by the segment supplied to its second input 66 which acts as a matte.
The result of the third transformation 32 is illustrated by the keyed segment 74 connected to an output 76 of the third transformation 32 via an arc 78. The keyed segment 74 is supplied to the sink 34 via an arc 80. As illustrated, the keyed segment 74 is the superposition of a portion of segment 56 over segment 26. The portion of segment 56 superposed corresponds to the portion which is in spacial correlation with segment 42.
One of the advantages of process node editing is that the operator has the ability to return to any previous node in the editing history of a sequence and make changes without having to redo subsequent steps which are automatically updated.
Another advantage is that the operator is able to specify many-to-many relationships between processes. For example, a single source could be shared between several different transformations such that a change to the source would be picked up by all the transformations simultaneously without any intervention from the user. For example, if source 24 is changed in FIG. 1, the entire process node graph illustrated will be modified accordingly.
A drawback of the process node architecture is the impossibility to manage the global timing of a sequence without a separate timeline editor.
Animation Curve Architectures
Animation curve editing typically involves a series of spline curves associated with a segment and placed on an x-y grid with each curve representing an animatable attribute of the segment. For example camera position, shininess, field of view, blurring, shadow, scaling, transparency, displacement, level, pan, and audio filtering are animatable attributes of video and/or audio segments.
FIG. 3 of the appended drawings shows a simple animation curve screen of an animation curve editor illustrating three spline curves 82, 84 and 86 of a segment.
As illustrated, the x-axis 88 of the x-y grid is calibrated in time units and the y-axis 90 is used as a gauge of intensity of the animatable attribute associated with the spline curves.
Each spline curve 82, 84 and 86 is formed of a plurality of vertexes 92 each indicating a keyframe value of the attribute represented by the curve. These vertexes are interconnected by interpolated lines 94. It is to be noted that the functions used to compute the interpolated lines 94 may vary.
A user may modify the shape of any of the spline curves to thereby modify the way the attribute controlled by this particular spline curve is applied to the segment.
One advantage of animation curve editing is that the user has detailed control over the timing and intensity of a wide range of attributes of the segment being composed. In the case of video, for example, control can be exercised over the spatial positioning of elements in a scene or their surface characteristics. In the case of audio, the volume level or panning, for example, can be controlled.
Producing a Multimedia Sequence
Some of the conventional editing packages found on the market provide separate editors each having an architecture adapted to handle one of the three types of editing generally described hereinabove.
To produce a multimedia sequence using one of these conventional editing packages, assuming that video and audio scenes have already been properly transferred to appropriate audio and video segments, a user has to perform many different manipulations.
The first step is to modify some of the animation curves of the segments to obtain the desired animated segments. This step is performed with the help of the animation curve editor.
The second step is to supply the animated segments to a process node editor to apply various effects to the animated segments. It is to be noted that, since there is no link between the animation curve editor and the process node editor, the user, after having done some transformations to the animated segment and wishing to modify the animation, must return to the animation editor and therefore must subsequently redo all the previously done transformations.
The third step is to supply the sink of the process node editor, containing the resulting transformed and animated segment to the timeline editor to be able to modify the timing of the segment and its position relative to other segments.
As previously discussed, the user, when noting that there is a problem with either the animation or the transformations, must return to the appropriate editor, correct the problem and then redo all the steps leading to the desired segment of the sequence.
The three steps must be repeated for each segment of a multimedia sequence.
It is to be noted that the user, wishing to modify the length of a segment of a multimedia sequence that has been done according to the three steps defined hereinabove, must return to the animation curve step to select a different length of data and then redo all the subsequent steps.
It is to be noted that the period of time involved with the steps of applying the sink of the process node editor to the timeline editor may be quite long with conventional process node editors that have to produce the entire processed segment from the animated segment and the applied transformations (a step usually called rendering).
Modified Timeline Architecture
A modified timeline architecture has been designed to benefit from some of the advantages of the process node architecture while keeping some of the advantages of the timeline architecture. It is to be noted that in current practice, these enhancements only apply to the editing of video segments; audio media editing is still handled according to the conventional timeline architecture described hereinabove.
FIG. 4 is a schematic view illustrating a display screen of a modified timeline editor. In a modified timeline editor, more than one video track 96, 98, 100 and 102, are stacked. The segments of the top video track 96 can be specified as transformed segments of the other video tracks 98, 100 and/or 102. To use this feature, a localized transformation is applied to a segment of the video tracks placed below the top video track 96 and in time synchronization with it.
For example, in FIG. 4, segment 104 may be specified as a localized transformation of segment 106. This localized transformation will take place only in the time range specified by segment 104. Therefore, the length of the transformed segment 104 may be modified by moving its beginning and/or end onto the video track 96.
If the segment 104 is moved elsewhere on the video track 96, the localized transformation will be applied to the new segments placed in time synchronisation with the new position of the segment 104.
Conceptually, in the modified timeline architecture, the video tracks 96-102 can be thought of as a stack of transparent layers arranged from top to bottom. The resulting sequence can be visualized by looking down from the top through the layers.
The modified timeline architecture has several drawbacks, for example:
inputs to a localized transformation are implicit and dependent on the position of the localized transformation relative to the other video tracks; this characteristic can complicate the moving of segments when the inputs must remain fixed; PA1 unlike in the conventional process node architecture, it is not possible to specify many-to-many relationships between segments; PA1 managing large process histories can become cumbersome since two neighboring segments on one video track can both derive their inputs from the track below; it is difficult for the user to focus in on the construction history of a specific segment; and PA1 the modified timeline architecture does not work well when the duration of a segment is modified by a transformation applied to it. PA1 at least one sequence track having a sequence node; the at least one sequence track being formed of a plurality of segments of multimedia related data; the sequence node including a sequence for ordering the plurality of segments in the sequence track; PA1 at least one process track having a process node; the process node being configured so as to determine the function of the process track; PA1 at least one scope control means removably attached to a process track and permanently attached to a sequence node; the scope control means transferring data from the process track to which it is removably attached to the sequence node to which it is permanently attached; PA1 playback device supplied with the at least one sequence track. PA1 forming at least one sequence track having a sequence node; the at least one sequence track being formed of a plurality of segments of multimedia related data; the sequence node including a sequence for ordering the plurality of segments in the sequence track; PA1 forming at least one process track having a process node; the process node being configured so as to determine the function of the process track; PA1 providing at least one scope control means removably attached to a process track and permanently attached to a sequence node; the scope control means transferring data from the process track to which it is removably attached to the sequence node to which it is permanently attached; PA1 supplying tue at least one sequence track to a playback device.